{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"NLU_multi_lingual_training_sentiment_classifier_demo_twitter.ipynb","provenance":[],"collapsed_sections":["zkufh760uvF3"]},"kernelspec":{"display_name":"Python 3","name":"python3"}},"cells":[{"cell_type":"markdown","metadata":{"id":"zkufh760uvF3"},"source":["![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)\n","\n","[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/nlu/blob/master/examples//colab/Training/multi_lingual/binary_text_classification/NLU_multi_lingual_training_sentiment_classifier_demo_twitter.ipynb)\n","\n","# Training a Sentiment Analysis Classifier with NLU \n","## 2 Class Twitter  Sentiment Classifier  Training\n","With the [SentimentDL model](https://nlp.johnsnowlabs.com/docs/en/annotators#sentimentdl-multi-class-sentiment-analysis-annotator) from Spark NLP you can achieve State Of the Art results on any multi class text classification problem \n","\n","This notebook showcases the following features : \n","\n","- How to train the deep learning classifier\n","- How to store a pipeline to disk\n","- How to load the pipeline from disk (Enables NLU offline mode)\n","\n","\n","*   List item\n","*   List item\n","\n","\n","You can achieve these results or even better on this dataset with training  data  : \n","\n","<br> \n","\n","![image.png]()\n","\n","You can achieve these results or even better on this dataset with test  data  : \n","\n","<br> \n","\n","![image.png]()"]},{"cell_type":"markdown","metadata":{"id":"dur2drhW5Rvi"},"source":["# 1. Install Java 8 and NLU"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"hFGnBCHavltY","executionInfo":{"status":"ok","timestamp":1620202068188,"user_tz":-300,"elapsed":106986,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"54e36062-3758-4478-9353-614909af18c6"},"source":["!wget https://setup.johnsnowlabs.com/nlu/colab.sh -O - | bash\n","  \n","\n","import nlu"],"execution_count":null,"outputs":[{"output_type":"stream","text":["--2021-05-05 08:06:01--  https://raw.githubusercontent.com/JohnSnowLabs/nlu/master/scripts/colab_setup.sh\n","Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.109.133, ...\n","Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.\n","HTTP request sent, awaiting response... 200 OK\n","Length: 1671 (1.6K) [text/plain]\n","Saving to: ‘STDOUT’\n","\n","\r-                     0%[                    ]       0  --.-KB/s               Installing  NLU 3.0.0 with  PySpark 3.0.2 and Spark NLP 3.0.1 for Google Colab ...\n","\r-                   100%[===================>]   1.63K  --.-KB/s    in 0.001s  \n","\n","2021-05-05 08:06:02 (1.81 MB/s) - written to stdout [1671/1671]\n","\n","\u001b[K     |████████████████████████████████| 204.8MB 76kB/s \n","\u001b[K     |████████████████████████████████| 153kB 48.2MB/s \n","\u001b[K     |████████████████████████████████| 204kB 21.6MB/s \n","\u001b[K     |████████████████████████████████| 204kB 49.6MB/s \n","\u001b[?25h  Building wheel for pyspark (setup.py) ... \u001b[?25l\u001b[?25hdone\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"f4KkTfnR5Ugg"},"source":["# 2. Download twitter Sentiment dataset \n","https://www.kaggle.com/cosmos98/twitter-and-reddit-sentimental-analysis-dataset\n","#Context\n","\n","This is was a Dataset Created as a part of the university Project On Sentimental Analysis On Multi-Source Social Media Platforms using PySpark."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"OrVb5ZMvvrQD","executionInfo":{"status":"ok","timestamp":1620202068746,"user_tz":-300,"elapsed":107458,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"4598d263-965a-47f3-9fc6-a3869e7b4318"},"source":["! wget http://ckl-it.de/wp-content/uploads/2021/02/twitter_data_multi_lang.csv\n"],"execution_count":null,"outputs":[{"output_type":"stream","text":["--2021-05-05 08:07:47--  http://ckl-it.de/wp-content/uploads/2021/02/twitter_data_multi_lang.csv\n","Resolving ckl-it.de (ckl-it.de)... 217.160.0.108, 2001:8d8:100f:f000::209\n","Connecting to ckl-it.de (ckl-it.de)|217.160.0.108|:80... connected.\n","HTTP request sent, awaiting response... 200 OK\n","Length: 125908 (123K) [text/csv]\n","Saving to: ‘twitter_data_multi_lang.csv’\n","\n","twitter_data_multi_ 100%[===================>] 122.96K   357KB/s    in 0.3s    \n","\n","2021-05-05 08:07:48 (357 KB/s) - ‘twitter_data_multi_lang.csv’ saved [125908/125908]\n","\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":419},"id":"y4xSRWIhwT28","executionInfo":{"status":"ok","timestamp":1620202069157,"user_tz":-300,"elapsed":107840,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"b98f130f-2334-4472-e9d5-585b3e4f0417"},"source":["import pandas as pd\n","train_path = '/content/twitter_data_multi_lang.csv'\n","\n","train_df = pd.read_csv(train_path)\n","train_df.test_sentences = train_df.test_sentences.astype(str)\n","# the text data to use for classification should be in a column named 'text'\n","# the label column must have name 'y' name be of type str\n","train_df= train_df[[\"text\",\"y\"]]\n","from sklearn.model_selection import train_test_split\n","train_df, test_df = train_test_split(train_df, test_size=0.2)\n","train_df"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>text</th>\n","      <th>y</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>261</th>\n","      <td>narender modi think the candidate who standing...</td>\n","      <td>negative</td>\n","    </tr>\n","    <tr>\n","      <th>250</th>\n","      <td>you forgot one more cruel incident happened ka...</td>\n","      <td>negative</td>\n","    </tr>\n","    <tr>\n","      <th>55</th>\n","      <td>modi the countryhe worked chaiwala not chowkid...</td>\n","      <td>negative</td>\n","    </tr>\n","    <tr>\n","      <th>105</th>\n","      <td>india wants leader who has vision success buil...</td>\n","      <td>positive</td>\n","    </tr>\n","    <tr>\n","      <th>532</th>\n","      <td>wrong assumptions made letter\\r\\nbeing modi fa...</td>\n","      <td>negative</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>168</th>\n","      <td>this hindutva terrorism enjoys the tacit suppo...</td>\n","      <td>positive</td>\n","    </tr>\n","    <tr>\n","      <th>80</th>\n","      <td>modis opposition trying defame him they not wa...</td>\n","      <td>negative</td>\n","    </tr>\n","    <tr>\n","      <th>560</th>\n","      <td>modi has crippled economy destroyed jobs far f...</td>\n","      <td>positive</td>\n","    </tr>\n","    <tr>\n","      <th>383</th>\n","      <td>and hope hindustan will over take all other co...</td>\n","      <td>negative</td>\n","    </tr>\n","    <tr>\n","      <th>133</th>\n","      <td>why not ask them vote for modern india modi wh...</td>\n","      <td>negative</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>480 rows × 2 columns</p>\n","</div>"],"text/plain":["                                                  text         y\n","261  narender modi think the candidate who standing...  negative\n","250  you forgot one more cruel incident happened ka...  negative\n","55   modi the countryhe worked chaiwala not chowkid...  negative\n","105  india wants leader who has vision success buil...  positive\n","532  wrong assumptions made letter\\r\\nbeing modi fa...  negative\n","..                                                 ...       ...\n","168  this hindutva terrorism enjoys the tacit suppo...  positive\n","80   modis opposition trying defame him they not wa...  negative\n","560  modi has crippled economy destroyed jobs far f...  positive\n","383  and hope hindustan will over take all other co...  negative\n","133  why not ask them vote for modern india modi wh...  negative\n","\n","[480 rows x 2 columns]"]},"metadata":{"tags":[]},"execution_count":3}]},{"cell_type":"markdown","metadata":{"id":"0296Om2C5anY"},"source":["# 3. Train Deep Learning Classifier using nlu.load('train.sentiment')\n","\n","You dataset label column should be named 'y' and the feature column with text data should be named 'text'"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":844},"id":"IKK_Ii_gjJfF","executionInfo":{"status":"ok","timestamp":1620202522330,"user_tz":-300,"elapsed":560988,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"5db3cc9e-7808-46d0-9f4e-934e1a3650ee"},"source":["trainable_pipe = nlu.load('xx.embed_sentence.labse train.sentiment')\n","# We need to train longer and user smaller LR for NON-USE based sentence embeddings usually\n","# We could tune the hyperparameters further with hyperparameter tuning methods like gridsearch\n","# Also longer training gives more accuracy\n","trainable_pipe['trainable_sentiment_dl'].setMaxEpochs(60)  \n","trainable_pipe['trainable_sentiment_dl'].setLr(0.005) \n","fitted_pipe = trainable_pipe.fit(train_df)\n","# predict with the trainable pipeline on dataset and get predictions\n","preds = fitted_pipe.predict(train_df,output_level='document')\n","\n","#sentence detector that is part of the pipe generates sone NaNs. lets drop them first\n","preds.dropna(inplace=True)\n","from sklearn.metrics import classification_report\n","print(classification_report(preds['y'], preds['sentiment']))\n","\n","preds"],"execution_count":null,"outputs":[{"output_type":"stream","text":["labse download started this may take some time.\n","Approximate size to download 1.7 GB\n","[OK!]\n","sentence_detector_dl download started this may take some time.\n","Approximate size to download 354.6 KB\n","[OK!]\n","              precision    recall  f1-score   support\n","\n","    negative       0.93      0.97      0.95       233\n","    positive       0.97      0.94      0.95       247\n","\n","    accuracy                           0.95       480\n","   macro avg       0.95      0.95      0.95       480\n","weighted avg       0.95      0.95      0.95       480\n","\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>y</th>\n","      <th>trained_sentiment</th>\n","      <th>text</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.997279</td>\n","      <td>narender modi think the candidate who standing...</td>\n","      <td>261</td>\n","      <td>negative</td>\n","      <td>negative</td>\n","      <td>narender modi think the candidate who standing...</td>\n","      <td>[narender modi think the candidate who standin...</td>\n","      <td>[-0.07268679887056351, 0.06004006788134575, 0....</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>0.999994</td>\n","      <td>you forgot one more cruel incident happened ka...</td>\n","      <td>250</td>\n","      <td>negative</td>\n","      <td>negative</td>\n","      <td>you forgot one more cruel incident happened ka...</td>\n","      <td>[you forgot one more cruel incident happened k...</td>\n","      <td>[0.07269874960184097, -0.027332717552781105, -...</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>0.999994</td>\n","      <td>modi the countryhe worked chaiwala not chowkid...</td>\n","      <td>55</td>\n","      <td>negative</td>\n","      <td>negative</td>\n","      <td>modi the countryhe worked chaiwala not chowkid...</td>\n","      <td>[modi the countryhe worked chaiwala not chowki...</td>\n","      <td>[0.016793066635727882, -0.021795757114887238, ...</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>1.000000</td>\n","      <td>india wants leader who has vision success buil...</td>\n","      <td>105</td>\n","      <td>positive</td>\n","      <td>positive</td>\n","      <td>india wants leader who has vision success buil...</td>\n","      <td>[india wants leader who has vision success bui...</td>\n","      <td>[-0.049463558942079544, 0.04899046570062637, 0...</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>0.999807</td>\n","      <td>wrong assumptions made letter being modi fan d...</td>\n","      <td>532</td>\n","      <td>negative</td>\n","      <td>negative</td>\n","      <td>wrong assumptions made letter\\r\\nbeing modi fa...</td>\n","      <td>[wrong assumptions made letter being modi fan ...</td>\n","      <td>[0.02427481859922409, 0.03929490968585014, -0....</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>475</th>\n","      <td>0.999200</td>\n","      <td>this hindutva terrorism enjoys the tacit suppo...</td>\n","      <td>168</td>\n","      <td>positive</td>\n","      <td>positive</td>\n","      <td>this hindutva terrorism enjoys the tacit suppo...</td>\n","      <td>[this hindutva terrorism enjoys the tacit supp...</td>\n","      <td>[0.05460929870605469, -0.00930434837937355, 0....</td>\n","    </tr>\n","    <tr>\n","      <th>476</th>\n","      <td>0.999818</td>\n","      <td>modis opposition trying defame him they not wa...</td>\n","      <td>80</td>\n","      <td>negative</td>\n","      <td>negative</td>\n","      <td>modis opposition trying defame him they not wa...</td>\n","      <td>[modis opposition trying defame him they not w...</td>\n","      <td>[-0.06659215688705444, 0.01988210715353489, 0....</td>\n","    </tr>\n","    <tr>\n","      <th>477</th>\n","      <td>0.999971</td>\n","      <td>modi has crippled economy destroyed jobs far f...</td>\n","      <td>560</td>\n","      <td>positive</td>\n","      <td>negative</td>\n","      <td>modi has crippled economy destroyed jobs far f...</td>\n","      <td>[modi has crippled economy destroyed jobs far ...</td>\n","      <td>[-0.0314302034676075, 0.027095980942249298, -0...</td>\n","    </tr>\n","    <tr>\n","      <th>478</th>\n","      <td>1.000000</td>\n","      <td>and hope hindustan will over take all other co...</td>\n","      <td>383</td>\n","      <td>negative</td>\n","      <td>positive</td>\n","      <td>and hope hindustan will over take all other co...</td>\n","      <td>[and hope hindustan will over take all other c...</td>\n","      <td>[0.048234350979328156, -0.037638068199157715, ...</td>\n","    </tr>\n","    <tr>\n","      <th>479</th>\n","      <td>0.998324</td>\n","      <td>why not ask them vote for modern india modi wh...</td>\n","      <td>133</td>\n","      <td>negative</td>\n","      <td>negative</td>\n","      <td>why not ask them vote for modern india modi wh...</td>\n","      <td>[why not ask them vote for modern india modi w...</td>\n","      <td>[-0.03812406584620476, 0.07693956047296524, -0...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>480 rows × 8 columns</p>\n","</div>"],"text/plain":["     trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                        0.997279  ...  [-0.07268679887056351, 0.06004006788134575, 0....\n","1                        0.999994  ...  [0.07269874960184097, -0.027332717552781105, -...\n","2                        0.999994  ...  [0.016793066635727882, -0.021795757114887238, ...\n","3                        1.000000  ...  [-0.049463558942079544, 0.04899046570062637, 0...\n","4                        0.999807  ...  [0.02427481859922409, 0.03929490968585014, -0....\n","..                            ...  ...                                                ...\n","475                      0.999200  ...  [0.05460929870605469, -0.00930434837937355, 0....\n","476                      0.999818  ...  [-0.06659215688705444, 0.01988210715353489, 0....\n","477                      0.999971  ...  [-0.0314302034676075, 0.027095980942249298, -0...\n","478                      1.000000  ...  [0.048234350979328156, -0.037638068199157715, ...\n","479                      0.998324  ...  [-0.03812406584620476, 0.07693956047296524, -0...\n","\n","[480 rows x 8 columns]"]},"metadata":{"tags":[]},"execution_count":4}]},{"cell_type":"markdown","metadata":{"id":"_1jxw3GnVGlI"},"source":["# 3.1 evaluate on Test Data"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Fxx4yNkNVGFl","executionInfo":{"status":"ok","timestamp":1620202543579,"user_tz":-300,"elapsed":582223,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"9f534cb3-736b-4d2f-f465-df0ccec6716b"},"source":["preds = fitted_pipe.predict(test_df,output_level='document')\n","\n","#sentence detector that is part of the pipe generates sone NaNs. lets drop them first\n","preds.dropna(inplace=True)\n","print(classification_report(preds['y'], preds['sentiment']))\n"],"execution_count":null,"outputs":[{"output_type":"stream","text":["              precision    recall  f1-score   support\n","\n","    negative       0.75      0.70      0.72        67\n","     neutral       0.00      0.00      0.00         0\n","    positive       0.65      0.68      0.67        53\n","\n","    accuracy                           0.69       120\n","   macro avg       0.47      0.46      0.46       120\n","weighted avg       0.71      0.69      0.70       120\n","\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"BD5OKO4Umc5U"},"source":["# 4. Test Model  on 20 languages!"]},{"cell_type":"code","metadata":{"id":"OQ72hP9unML7","colab":{"base_uri":"https://localhost:8080/","height":776},"executionInfo":{"status":"ok","timestamp":1620202562769,"user_tz":-300,"elapsed":601405,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"409a40af-dff1-4910-8136-c74f09527aa5"},"source":["train_df = pd.read_csv(\"/content/twitter_data_multi_lang.csv\")\n","preds = fitted_pipe.predict(train_df[[\"test_sentences\",\"y\"]].iloc[:100],output_level='document')\n","\n","#sentence detector that is part of the pipe generates sone NaNs. lets drop them first\n","preds.dropna(inplace=True)\n","print(classification_report(preds['y'], preds['sentiment']))\n","\n","\n","preds"],"execution_count":null,"outputs":[{"output_type":"stream","text":["              precision    recall  f1-score   support\n","\n","    negative       0.83      0.82      0.82        49\n","     neutral       0.00      0.00      0.00         0\n","    positive       0.85      0.80      0.83        51\n","\n","    accuracy                           0.81       100\n","   macro avg       0.56      0.54      0.55       100\n","weighted avg       0.84      0.81      0.83       100\n","\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>y</th>\n","      <th>trained_sentiment</th>\n","      <th>text</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.999401</td>\n","      <td>how narendra modi has almost killed the indian...</td>\n","      <td>0</td>\n","      <td>negative</td>\n","      <td>negative</td>\n","      <td>how narendra modi has almost killed the indian...</td>\n","      <td>[how narendra modi has almost killed the india...</td>\n","      <td>[-0.05593567714095116, 0.050420816987752914, -...</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>0.995770</td>\n","      <td>تعتقد أنه كان مودي وراء هذا الحادث</td>\n","      <td>1</td>\n","      <td>negative</td>\n","      <td>negative</td>\n","      <td>تعتقد أنه كان مودي وراء هذا الحادث</td>\n","      <td>[تعتقد أنه كان مودي وراء هذا الحادث]</td>\n","      <td>[0.007358793169260025, -0.0520767867565155, 0....</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>0.998129</td>\n","      <td>カマル・ハサーンがチョウキダール・モディを連れて行くカマル・ハサーン・モディの金持ちが貧しい...</td>\n","      <td>2</td>\n","      <td>negative</td>\n","      <td>negative</td>\n","      <td>カマル・ハサーンがチョウキダール・モディを連れて行くカマル・ハサーン・モディの金持ちが貧しい...</td>\n","      <td>[カマル・ハサーンがチョウキダール・モディを連れて行くカマル・ハサーン・モディの金持ちが貧し...</td>\n","      <td>[-0.012155424803495407, -0.02065391279757023, ...</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>0.998547</td>\n","      <td>связанное имя с фамилией, а не bcz религия, св...</td>\n","      <td>3</td>\n","      <td>negative</td>\n","      <td>negative</td>\n","      <td>связанное имя с фамилией, а не bcz религия, св...</td>\n","      <td>[связанное имя с фамилией, а не bcz религия, с...</td>\n","      <td>[-0.006620907690376043, 0.02574392780661583, -...</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>1.000000</td>\n","      <td>kdokoli lepší než modi, když nehruji vypršela,...</td>\n","      <td>4</td>\n","      <td>positive</td>\n","      <td>positive</td>\n","      <td>kdokoli lepší než modi, když nehruji vypršela,...</td>\n","      <td>[kdokoli lepší než modi, když nehruji vypršela...</td>\n","      <td>[-0.04917769134044647, 0.017523039132356644, -...</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>95</th>\n","      <td>0.999987</td>\n","      <td>lol qui va épouser son hippopotame tous les ho...</td>\n","      <td>95</td>\n","      <td>positive</td>\n","      <td>positive</td>\n","      <td>lol qui va épouser son hippopotame tous les ho...</td>\n","      <td>[lol qui va épouser son hippopotame tous les h...</td>\n","      <td>[-0.01001912634819746, -0.031715311110019684, ...</td>\n","    </tr>\n","    <tr>\n","      <th>96</th>\n","      <td>0.999994</td>\n","      <td>拉贾斯坦邦州长卡莉安·辛格·阿里加3月23日全都是bjp工人，希望bjp胜利，希望莫迪再次成...</td>\n","      <td>96</td>\n","      <td>positive</td>\n","      <td>positive</td>\n","      <td>拉贾斯坦邦州长卡莉安·辛格·阿里加3月23日全都是bjp工人，希望bjp胜利，希望莫迪再次成...</td>\n","      <td>[拉贾斯坦邦州长卡莉安·辛格·阿里加3月23日全都是bjp工人，希望bjp胜利，希望莫迪再次...</td>\n","      <td>[0.009000560268759727, -0.021888386458158493, ...</td>\n","    </tr>\n","    <tr>\n","      <th>97</th>\n","      <td>0.979382</td>\n","      <td>మోడీ భక్తులు రాహుల్ గురించి అబద్ధాలు చెబుతున్న...</td>\n","      <td>97</td>\n","      <td>positive</td>\n","      <td>positive</td>\n","      <td>మోడీ భక్తులు రాహుల్ గురించి అబద్ధాలు చెబుతున్న...</td>\n","      <td>[మోడీ భక్తులు రాహుల్ గురించి అబద్ధాలు చెబుతున్...</td>\n","      <td>[-0.05518203601241112, -0.0041709886863827705,...</td>\n","    </tr>\n","    <tr>\n","      <th>98</th>\n","      <td>0.882261</td>\n","      <td>lol neha, je to jako dát hlavu zabít těm, kteř...</td>\n","      <td>98</td>\n","      <td>positive</td>\n","      <td>positive</td>\n","      <td>lol neha, je to jako dát hlavu zabít těm, kteř...</td>\n","      <td>[lol neha, je to jako dát hlavu zabít těm, kte...</td>\n","      <td>[-0.019701892510056496, -0.01936856471002102, ...</td>\n","    </tr>\n","    <tr>\n","      <th>99</th>\n","      <td>0.999997</td>\n","      <td>por favor venda nuestro bosque por favor haga ...</td>\n","      <td>99</td>\n","      <td>positive</td>\n","      <td>positive</td>\n","      <td>por favor venda nuestro bosque por favor haga ...</td>\n","      <td>[por favor venda nuestro bosque por favor haga...</td>\n","      <td>[-0.039666254073381424, -0.0194801464676857, -...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>100 rows × 8 columns</p>\n","</div>"],"text/plain":["    trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                       0.999401  ...  [-0.05593567714095116, 0.050420816987752914, -...\n","1                       0.995770  ...  [0.007358793169260025, -0.0520767867565155, 0....\n","2                       0.998129  ...  [-0.012155424803495407, -0.02065391279757023, ...\n","3                       0.998547  ...  [-0.006620907690376043, 0.02574392780661583, -...\n","4                       1.000000  ...  [-0.04917769134044647, 0.017523039132356644, -...\n","..                           ...  ...                                                ...\n","95                      0.999987  ...  [-0.01001912634819746, -0.031715311110019684, ...\n","96                      0.999994  ...  [0.009000560268759727, -0.021888386458158493, ...\n","97                      0.979382  ...  [-0.05518203601241112, -0.0041709886863827705,...\n","98                      0.882261  ...  [-0.019701892510056496, -0.01936856471002102, ...\n","99                      0.999997  ...  [-0.039666254073381424, -0.0194801464676857, -...\n","\n","[100 rows x 8 columns]"]},"metadata":{"tags":[]},"execution_count":6}]},{"cell_type":"markdown","metadata":{"id":"RjtuNUcvuJTT"},"source":["# The Model understands Englsih\n","![en](https://www.worldometers.info/img/flags/small/tn_nz-flag.gif)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"o0vu7PaWkcI7","executionInfo":{"status":"ok","timestamp":1620202564814,"user_tz":-300,"elapsed":603442,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"c5dd75ba-92bc-45b9-f6da-e44f6a90b35e"},"source":["fitted_pipe.predict(\"Congress's new policies made many people sad \")\n"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.971534</td>\n","      <td>Congress's new policies made many people sad</td>\n","      <td>0</td>\n","      <td>negative</td>\n","      <td>[Congress's new policies made many people sad]</td>\n","      <td>[0.004380704369395971, -0.00210917298682034, -...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.971534  ...  [0.004380704369395971, -0.00210917298682034, -...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":7}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"1ykjRQhCtQ4w","executionInfo":{"status":"ok","timestamp":1620202564816,"user_tz":-300,"elapsed":603426,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"1bfa4d52-54a2-43e3-99ed-b054129c8b41"},"source":["fitted_pipe.predict(\"Congress's new policies made many people happy \")\n"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.999984</td>\n","      <td>Congress's new policies made many people happy</td>\n","      <td>0</td>\n","      <td>positive</td>\n","      <td>[Congress's new policies made many people happy]</td>\n","      <td>[0.025979558005928993, -0.007445275783538818, ...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.999984  ...  [0.025979558005928993, -0.007445275783538818, ...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":8}]},{"cell_type":"markdown","metadata":{"id":"vohym-XbuNHn"},"source":["# The Model understands German\n","![de](https://www.worldometers.info/img/flags/small/tn_gm-flag.gif)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"dzaaZrI4tVWc","executionInfo":{"status":"ok","timestamp":1620202566199,"user_tz":-300,"elapsed":604799,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"7a0eeb2c-0d28-4bfa-fe8b-a5827afe89f1"},"source":["# German for: 'Congress's newest polices made many people poor, sad and depressed '\n","fitted_pipe.predict(\"Die neue Politik des Kongresses machte viele Menschen arm, traurig und depressiv \")\n"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.779711</td>\n","      <td>Die neue Politik des Kongresses machte viele M...</td>\n","      <td>0</td>\n","      <td>negative</td>\n","      <td>[Die neue Politik des Kongresses machte viele ...</td>\n","      <td>[-0.02746928110718727, 0.015148899517953396, -...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.779711  ...  [-0.02746928110718727, 0.015148899517953396, -...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":9}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"BbhgTSBGtTtJ","executionInfo":{"status":"ok","timestamp":1620202566201,"user_tz":-300,"elapsed":604792,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"9f0c244c-834a-4b01-8d60-df278e60de56"},"source":["# German for: 'Congress's newest polices made many people happy '\n","fitted_pipe.predict(\"Die neue Politik des Kongresses machte viele Menschen glücklich \")\n"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.999974</td>\n","      <td>Die neue Politik des Kongresses machte viele M...</td>\n","      <td>0</td>\n","      <td>positive</td>\n","      <td>[Die neue Politik des Kongresses machte viele ...</td>\n","      <td>[0.008141197264194489, -0.009829358197748661, ...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.999974  ...  [0.008141197264194489, -0.009829358197748661, ...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":10}]},{"cell_type":"markdown","metadata":{"id":"a1JbtmWquQwj"},"source":["# The Model understands Chinese\n","![zh](https://www.worldometers.info/img/flags/small/tn_ch-flag.gif)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"kYSYqtoRtc-P","executionInfo":{"status":"ok","timestamp":1620202567220,"user_tz":-300,"elapsed":605804,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"45ea8add-578f-4c7a-bf50-f5649f2b3bbe"},"source":["# Chinese for: 'Congress's newest polices made many people happy '\n","fitted_pipe.predict(\"国会的新政策使许多人感到高兴 \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.999998</td>\n","      <td>国会的新政策使许多人感到高兴</td>\n","      <td>0</td>\n","      <td>positive</td>\n","      <td>[国会的新政策使许多人感到高兴]</td>\n","      <td>[0.009464389644563198, -0.012016313150525093, ...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.999998  ...  [0.009464389644563198, -0.012016313150525093, ...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":11}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"06v9SD-QtlBU","executionInfo":{"status":"ok","timestamp":1620202567221,"user_tz":-300,"elapsed":605796,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"0e1f7d04-90d5-4765-a4d4-09d88e7ff62b"},"source":["# Chinese for: 'Congress's newest polices made many people poor, sad and depressed '\n","fitted_pipe.predict(\"国会的新政策使许多人变得贫穷，悲伤和沮丧 \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.998916</td>\n","      <td>国会的新政策使许多人变得贫穷，悲伤和沮丧</td>\n","      <td>0</td>\n","      <td>negative</td>\n","      <td>[国会的新政策使许多人变得贫穷，悲伤和沮丧]</td>\n","      <td>[-0.05506608635187149, -0.002640362363308668, ...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.998916  ...  [-0.05506608635187149, -0.002640362363308668, ...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":12}]},{"cell_type":"markdown","metadata":{"id":"9h7CvN4uu9Pb"},"source":["# Model understands Afrikaans\n","\n","![af](https://www.worldometers.info/img/flags/small/tn_sf-flag.gif)\n","\n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"VMPhbgw9twtf","executionInfo":{"status":"ok","timestamp":1620202568514,"user_tz":-300,"elapsed":607077,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"54ce4a9a-52dc-477c-8195-3fb28a321074"},"source":["#  Afrikaans for: 'Congress's newest polices made many people poor, sad and depressed '\n","fitted_pipe.predict(\"Die Kongres se nuwe beleid het baie mense arm, hartseer en depressief gemaak \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.720446</td>\n","      <td>Die Kongres se nuwe beleid het baie mense arm,...</td>\n","      <td>0</td>\n","      <td>negative</td>\n","      <td>[Die Kongres se nuwe beleid het baie mense arm...</td>\n","      <td>[-0.023684442043304443, 0.0034083062782883644,...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.720446  ...  [-0.023684442043304443, 0.0034083062782883644,...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":13}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"zWgNTIdkumhX","executionInfo":{"status":"ok","timestamp":1620202569943,"user_tz":-300,"elapsed":608497,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"7f7ccf28-2d4d-487d-9205-1c469d23275b"},"source":["#  Afrikaans for: 'Congress's newest polices made many people happy '\n","fitted_pipe.predict(\"Die Kongres se nuwe beleid het baie mense gelukkig gemaak \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.999999</td>\n","      <td>Die Kongres se nuwe beleid het baie mense gelu...</td>\n","      <td>0</td>\n","      <td>positive</td>\n","      <td>[Die Kongres se nuwe beleid het baie mense gel...</td>\n","      <td>[0.005836560856550932, -0.02982638217508793, -...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.999999  ...  [0.005836560856550932, -0.02982638217508793, -...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":14}]},{"cell_type":"markdown","metadata":{"id":"IlkmAaMoxTuy"},"source":["# The model understands Japanese\n","![ja](https://www.worldometers.info/img/flags/small/tn_ja-flag.gif)\n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":97},"id":"1IfJu3q8wwUt","executionInfo":{"status":"ok","timestamp":1620202569947,"user_tz":-300,"elapsed":608494,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"e1be5a26-606f-4bda-8494-ce0f58aabf14"},"source":["# Japanese for: 'Congress's newest polices made many people poor, sad and depressed '\n","fitted_pipe.predict(\"議会の新しい政策は多くの人々を貧しく、悲しくそして落ち込んだものにしました \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.993153</td>\n","      <td>議会の新しい政策は多くの人々を貧しく、悲しくそして落ち込んだものにしました</td>\n","      <td>0</td>\n","      <td>negative</td>\n","      <td>[議会の新しい政策は多くの人々を貧しく、悲しくそして落ち込んだものにしました]</td>\n","      <td>[-0.04006955772638321, 0.0033476136159151793, ...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.993153  ...  [-0.04006955772638321, 0.0033476136159151793, ...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":15}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"h3k7_PFhxOve","executionInfo":{"status":"ok","timestamp":1620202570964,"user_tz":-300,"elapsed":609500,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"76551d94-8d73-4904-9636-e1495852e33d"},"source":["\n","\t\t\n","# Japanese for: 'Congress's newest polices made many people happy '\n","fitted_pipe.predict(\"議会の新しい政策は多くの人々を幸せにしました \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.999996</td>\n","      <td>議会の新しい政策は多くの人々を幸せにしました</td>\n","      <td>0</td>\n","      <td>positive</td>\n","      <td>[議会の新しい政策は多くの人々を幸せにしました]</td>\n","      <td>[-0.017957264557480812, -0.015919487923383713,...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.999996  ...  [-0.017957264557480812, -0.015919487923383713,...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":16}]},{"cell_type":"markdown","metadata":{"id":"VGVvzl_30a0T"},"source":["# The  Model understands Turkish\n","![tr](https://www.worldometers.info/img/flags/small/tn_tu-flag.gif)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"DRNnuEeQz2pd","executionInfo":{"status":"ok","timestamp":1620202571532,"user_tz":-300,"elapsed":610061,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"515337da-dcec-4345-f112-cd35e644c79e"},"source":["#  Turkish for: 'Congress's newest polices made many people poor, sad and depressed '\n","fitted_pipe.predict(\"Kongrenin yeni politikaları birçok insanı fakir, hüzünlü ve depresif hale getirdi \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.995125</td>\n","      <td>Kongrenin yeni politikaları birçok insanı faki...</td>\n","      <td>0</td>\n","      <td>negative</td>\n","      <td>[Kongrenin yeni politikaları birçok insanı fak...</td>\n","      <td>[-0.02755207195878029, 0.012688503600656986, -...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.995125  ...  [-0.02755207195878029, 0.012688503600656986, -...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":17}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"aOSsiK6J0jWs","executionInfo":{"status":"ok","timestamp":1620202572302,"user_tz":-300,"elapsed":610810,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"45f525b8-617b-4469-d61e-df03469b8734"},"source":["#  Turkish for: 'Congress's newest polices made many people happy '\n","fitted_pipe.predict(\"Kongrenin yeni politikaları birçok insanı mutlu etti \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.999997</td>\n","      <td>Kongrenin yeni politikaları birçok insanı mutl...</td>\n","      <td>0</td>\n","      <td>positive</td>\n","      <td>[Kongrenin yeni politikaları birçok insanı mut...</td>\n","      <td>[0.019367843866348267, -0.0063224597834050655,...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.999997  ...  [0.019367843866348267, -0.0063224597834050655,...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":18}]},{"cell_type":"markdown","metadata":{"id":"803qL2gt0vlb"},"source":["#  The Model understands Hebrew\n","![he](https://www.worldometers.info/img/flags/small/tn_sf-flag.gif)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"XQ5VCtxw0pc0","executionInfo":{"status":"ok","timestamp":1620202573308,"user_tz":-300,"elapsed":611804,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"e9a8ce71-dbad-46b6-e556-f51b2c7ef208"},"source":["# Hebrew for: 'Congress's newest polices made many people poor, sad and depressed '\n","fitted_pipe.predict(\"המדיניות החדשה של הקונגרס גרמה לאנשים רבים להיות עניים, עצובים ומדוכאים \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.994727</td>\n","      <td>המדיניות החדשה של הקונגרס גרמה לאנשים רבים להי...</td>\n","      <td>0</td>\n","      <td>negative</td>\n","      <td>[המדיניות החדשה של הקונגרס גרמה לאנשים רבים לה...</td>\n","      <td>[-0.03273192420601845, -0.016592275351285934, ...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.994727  ...  [-0.03273192420601845, -0.016592275351285934, ...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":19}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"9w2ZHfns05A4","executionInfo":{"status":"ok","timestamp":1620202574066,"user_tz":-300,"elapsed":612547,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"cb13aabf-b93c-4fa1-e729-af42f241366b"},"source":["# Hebrew for: 'Congress's newest polices made many people happy '\n","fitted_pipe.predict(\"המדיניות החדשה של הקונגרס שימחה אנשים רבים \")\n","\t\t"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.999977</td>\n","      <td>המדיניות החדשה של הקונגרס שימחה אנשים רבים</td>\n","      <td>0</td>\n","      <td>positive</td>\n","      <td>[המדיניות החדשה של הקונגרס שימחה אנשים רבים]</td>\n","      <td>[0.0014839837094768882, -0.01997891254723072, ...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.999977  ...  [0.0014839837094768882, -0.01997891254723072, ...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":20}]},{"cell_type":"markdown","metadata":{"id":"SDlpd33H1HIX"},"source":["# The Model understands Telugu\n","![te](https://www.worldometers.info/img/flags/small/tn_in-flag.gif)\n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"Kc5n1bzv1BJT","executionInfo":{"status":"ok","timestamp":1620202574942,"user_tz":-300,"elapsed":613408,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"3a47d3cd-1748-4725-d6b2-329c6b3b8664"},"source":["# Telugu for: 'Congress's newest polices made many people poor, sad and depressed '\n","fitted_pipe.predict(\"కాంగ్రెస్ కొత్త విధానాలు చాలా మందిని పేదలుగా, విచారంగా, నిరాశకు గురి చేశాయి \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.994245</td>\n","      <td>కాంగ్రెస్ కొత్త విధానాలు చాలా మందిని పేదలుగా, ...</td>\n","      <td>0</td>\n","      <td>negative</td>\n","      <td>[కాంగ్రెస్ కొత్త విధానాలు చాలా మందిని పేదలుగా,...</td>\n","      <td>[-0.02907465025782585, -0.02225475199520588, -...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.994245  ...  [-0.02907465025782585, -0.02225475199520588, -...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":21}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"-l-u6vrz1Obe","executionInfo":{"status":"ok","timestamp":1620202575499,"user_tz":-300,"elapsed":613956,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"24fcaacc-949f-4243-d9ba-02537d895c71"},"source":["# Telugu for: 'Congress's newest polices made many people happy '\n","fitted_pipe.predict(\"కాంగ్రెస్ కొత్త విధానాలు చాలా మందికి సంతోషాన్నిచ్చాయి \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>1.0</td>\n","      <td>కాంగ్రెస్ కొత్త విధానాలు చాలా మందికి సంతోషాన్న...</td>\n","      <td>0</td>\n","      <td>positive</td>\n","      <td>[కాంగ్రెస్ కొత్త విధానాలు చాలా మందికి సంతోషాన్...</td>\n","      <td>[0.003831395646557212, -0.034895412623882294, ...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                           1.0  ...  [0.003831395646557212, -0.034895412623882294, ...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":22}]},{"cell_type":"markdown","metadata":{"id":"nziBUe8t1Zwn"},"source":["# Model understands Russian\n","![ru](https://www.worldometers.info/img/flags/small/tn_rs-flag.gif)\n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"Ckyjl3YQ1VFn","executionInfo":{"status":"ok","timestamp":1620202576540,"user_tz":-300,"elapsed":614990,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"45003fc1-6c5e-4ca2-9ffa-762379c6e612"},"source":["#  Russian for: 'Congress's newest polices made many people poor, sad and depressed '\n","fitted_pipe.predict(\"Новая политика Конгресса сделала многих людей бедными, грустными и подавленными \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.997476</td>\n","      <td>Новая политика Конгресса сделала многих людей ...</td>\n","      <td>0</td>\n","      <td>negative</td>\n","      <td>[Новая политика Конгресса сделала многих людей...</td>\n","      <td>[-0.029941784217953682, 0.016272399574518204, ...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.997476  ...  [-0.029941784217953682, 0.016272399574518204, ...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":23}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"GIdWkfGv1gFz","executionInfo":{"status":"ok","timestamp":1620202577054,"user_tz":-300,"elapsed":615494,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"ab04d8b1-3682-4a20-87bb-e0cb1ff12446"},"source":["\n","\t\t\n","#  Russian for: 'Congress's newest polices made many people happy '\n","fitted_pipe.predict(\"Новая политика Конгресса порадовала многих людей \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.999979</td>\n","      <td>Новая политика Конгресса порадовала многих людей</td>\n","      <td>0</td>\n","      <td>positive</td>\n","      <td>[Новая политика Конгресса порадовала многих лю...</td>\n","      <td>[-0.002074694959446788, 0.014204198494553566, ...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.999979  ...  [-0.002074694959446788, 0.014204198494553566, ...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":24}]},{"cell_type":"markdown","metadata":{"id":"8R1j9mwz2Cm4"},"source":["# Model understands Urdu\n","![ur](https://www.worldometers.info/img/flags/small/tn_pk-flag.gif)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"j4zwvRV11pcG","executionInfo":{"status":"ok","timestamp":1620202577597,"user_tz":-300,"elapsed":616025,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"9e08445b-75e1-4d68-b34b-98a6c90d7201"},"source":["\n","\t\t\n","# Urdu for: 'Congress's newest polices made many people poor, sad and depressed '\n","fitted_pipe.predict(\"کانگریس کی نئی پالیسیوں نے بہت سارے لوگوں کو غریب ، افسردہ اور افسردہ کردیا \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.999625</td>\n","      <td>کانگریس کی نئی پالیسیوں نے بہت سارے لوگوں کو غ...</td>\n","      <td>0</td>\n","      <td>negative</td>\n","      <td>[کانگریس کی نئی پالیسیوں نے بہت سارے لوگوں کو ...</td>\n","      <td>[-0.032778408378362656, -0.01915016397833824, ...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.999625  ...  [-0.032778408378362656, -0.01915016397833824, ...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":25}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"SxzTuK4b2UKV","executionInfo":{"status":"ok","timestamp":1620202578836,"user_tz":-300,"elapsed":617254,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"cb327854-4b6d-4179-964b-db657342db20"},"source":["# Urdu for: 'Congress's newest polices made many people happy '\n","fitted_pipe.predict(\"کانگریس کی نئی پالیسیوں نے بہت سارے لوگوں کو خوش کیا \")\n","\t\t"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.995908</td>\n","      <td>کانگریس کی نئی پالیسیوں نے بہت سارے لوگوں کو خ...</td>\n","      <td>0</td>\n","      <td>positive</td>\n","      <td>[کانگریس کی نئی پالیسیوں نے بہت سارے لوگوں کو ...</td>\n","      <td>[0.0033543300814926624, -0.0338786281645298, -...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.995908  ...  [0.0033543300814926624, -0.0338786281645298, -...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":26}]},{"cell_type":"markdown","metadata":{"id":"RoNg-C3k1qcX"},"source":["# Model understands Hindi\n","![hi](https://www.worldometers.info/img/flags/small/tn_in-flag.gif)\n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"QZ9RT5Wv1r1n","executionInfo":{"status":"ok","timestamp":1620202579420,"user_tz":-300,"elapsed":617832,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"17e2dba5-3371-40dd-b831-9148fa4c7007"},"source":["# hindi for: 'Congress's newest polices made many people poor, sad and depressed '\n","fitted_pipe.predict(\"कांग्रेस की नई नीतियों ने कई लोगों को गरीब, दुखी और उदास बना दिया \")\n","\t\t"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.995045</td>\n","      <td>कांग्रेस की नई नीतियों ने कई लोगों को गरीब, दु...</td>\n","      <td>0</td>\n","      <td>negative</td>\n","      <td>[कांग्रेस की नई नीतियों ने कई लोगों को गरीब, द...</td>\n","      <td>[-0.030935177579522133, -0.011918646283447742,...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.995045  ...  [-0.030935177579522133, -0.011918646283447742,...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":27}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"quM-IL2i12-B","executionInfo":{"status":"ok","timestamp":1620202580321,"user_tz":-300,"elapsed":618728,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"4ef407db-efe4-4ef8-e2b6-2eeb63ab775e"},"source":["# hindi for: 'Congress's newest polices made many people happy '\n","fitted_pipe.predict(\"कांग्रेस की नई नीतियों ने कई लोगों को खुश किया \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.999983</td>\n","      <td>कांग्रेस की नई नीतियों ने कई लोगों को खुश किया</td>\n","      <td>0</td>\n","      <td>positive</td>\n","      <td>[कांग्रेस की नई नीतियों ने कई लोगों को खुश किया]</td>\n","      <td>[0.005392681807279587, -0.024082284420728683, ...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.999983  ...  [0.005392681807279587, -0.024082284420728683, ...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":28}]},{"cell_type":"markdown","metadata":{"id":"HKj5yWwwMplH"},"source":["# The Model understands French\n","![fr](https://www.worldometers.info/img/flags/small/tn_fr-flag.gif)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"CUHcJZfJMplL","executionInfo":{"status":"ok","timestamp":1620202582298,"user_tz":-300,"elapsed":620676,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"41c1b052-da38-40be-860a-b006f2f9df6e"},"source":["\t\t\n","# French for: 'Congress's newest polices made many people poor, sad and depressed '\n","fitted_pipe.predict(\"Les nouvelles politiques du Congrès ont rendu de nombreuses personnes pauvres, tristes et déprimées \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.995686</td>\n","      <td>Les nouvelles politiques du Congrès ont rendu ...</td>\n","      <td>0</td>\n","      <td>negative</td>\n","      <td>[Les nouvelles politiques du Congrès ont rendu...</td>\n","      <td>[-0.017834002152085304, 0.011118772439658642, ...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.995686  ...  [-0.017834002152085304, 0.011118772439658642, ...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":31}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"57NY2XoTMplM","executionInfo":{"status":"ok","timestamp":1620202582866,"user_tz":-300,"elapsed":621235,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"4fa2fd7c-9697-4b60-c270-b0c2e644b020"},"source":["# French for: 'Congress's newest polices made many people happy '\n","fitted_pipe.predict(\"Les nouvelles politiques du Congrès ont rendu de nombreuses personnes heureuses \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.999999</td>\n","      <td>Les nouvelles politiques du Congrès ont rendu ...</td>\n","      <td>0</td>\n","      <td>positive</td>\n","      <td>[Les nouvelles politiques du Congrès ont rendu...</td>\n","      <td>[0.019515689462423325, -0.010051749646663666, ...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.999999  ...  [0.019515689462423325, -0.010051749646663666, ...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":32}]},{"cell_type":"markdown","metadata":{"id":"jD2TBgT0Nq6F"},"source":["# The Model understands Thai\n","![th](https://www.worldometers.info/img/flags/small/tn_th-flag.gif)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"gBp11S5GNq6S","executionInfo":{"status":"ok","timestamp":1620202584197,"user_tz":-300,"elapsed":622559,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"878c5149-214e-427a-d206-325ad89d6e2c"},"source":["\t\t\n","# Thai for: 'Congress's newest polices made many people poor, sad and depressed '\n","fitted_pipe.predict(\"นโยบายใหม่ของสภาคองเกรสทำให้หลายคนยากจนเศร้าและหดหู่ \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.727099</td>\n","      <td>นโยบายใหม่ของสภาคองเกรสทำให้หลายคนยากจนเศร้าแล...</td>\n","      <td>0</td>\n","      <td>positive</td>\n","      <td>[นโยบายใหม่ของสภาคองเกรสทำให้หลายคนยากจนเศร้าแ...</td>\n","      <td>[4.369320549812983e-07, -0.002880594925954938,...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.727099  ...  [4.369320549812983e-07, -0.002880594925954938,...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":33}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"KgatiiyuZumz","executionInfo":{"status":"ok","timestamp":1620202584199,"user_tz":-300,"elapsed":622549,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"89c4d1ab-b36b-4e97-a414-1d07866c3304"},"source":["# Thai for: 'Congress's newest polices made many people happy '\n","fitted_pipe.predict(\"นโยบายใหม่ของสภาคองเกรสทำให้หลายคนพอใจ \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.999999</td>\n","      <td>นโยบายใหม่ของสภาคองเกรสทำให้หลายคนพอใจ</td>\n","      <td>0</td>\n","      <td>positive</td>\n","      <td>[นโยบายใหม่ของสภาคองเกรสทำให้หลายคนพอใจ]</td>\n","      <td>[0.012098133563995361, 0.006513903383165598, -...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.999999  ...  [0.012098133563995361, 0.006513903383165598, -...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":34}]},{"cell_type":"markdown","metadata":{"id":"mLItI4KZOElB"},"source":["# The Model understands Khmer\n","![km](https://www.worldometers.info/img/flags/small/tn_cb-flag.gif)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"Fxh1gasROElC","executionInfo":{"status":"ok","timestamp":1620202585353,"user_tz":-300,"elapsed":623695,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"9c517c18-1b44-45a5-8941-a1c21c21a454"},"source":["# Khmer for: 'Congress's newest polices made many people poor, sad and depressed '\n","fitted_pipe.predict(\"គោលនយោបាយថ្មីរបស់សភាបានធ្វើឱ្យប្រជាជនជាច្រើនក្រីក្រក្រៀមក្រំនិងធ្លាក់ទឹកចិត្ត \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.987534</td>\n","      <td>គោលនយោបាយថ្មីរបស់សភាបានធ្វើឱ្យប្រជាជនជាច្រើនក្...</td>\n","      <td>0</td>\n","      <td>negative</td>\n","      <td>[គោលនយោបាយថ្មីរបស់សភាបានធ្វើឱ្យប្រជាជនជាច្រើនក...</td>\n","      <td>[-0.045212410390377045, 0.010355290956795216, ...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.987534  ...  [-0.045212410390377045, 0.010355290956795216, ...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":35}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"SWbqMgAwOElC","executionInfo":{"status":"ok","timestamp":1620202586047,"user_tz":-300,"elapsed":624382,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"922c462d-f20b-4501-dbcf-6368801b889f"},"source":["# Khmer for: 'Congress's newest polices made many people happy '\n","fitted_pipe.predict(\"គោលនយោបាយថ្មីរបស់សភាបានធ្វើឱ្យមនុស្សជាច្រើនសប្បាយរីករាយ \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.999997</td>\n","      <td>គោលនយោបាយថ្មីរបស់សភាបានធ្វើឱ្យមនុស្សជាច្រើនសប្...</td>\n","      <td>0</td>\n","      <td>positive</td>\n","      <td>[គោលនយោបាយថ្មីរបស់សភាបានធ្វើឱ្យមនុស្សជាច្រើនសប...</td>\n","      <td>[-0.025576695799827576, -0.020313698798418045,...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.999997  ...  [-0.025576695799827576, -0.020313698798418045,...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":36}]},{"cell_type":"markdown","metadata":{"id":"lvE-LbNiPoBT"},"source":["# The Model understands Yiddish\n","![yi](https://www.worldometers.info/img/flags/small/tn_pl-flag.gif)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"sZlmLhajPoBb","executionInfo":{"status":"ok","timestamp":1620202586621,"user_tz":-300,"elapsed":624922,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"3715ec9a-21a3-4375-a8cb-7ea846674419"},"source":["\t\t\n","# Yiddish for: 'Congress's newest polices made many people poor, sad and depressed '\n","fitted_pipe.predict(\"קאָנגרעס ס נייַ פּאַלאַסיז געמאכט פילע מענטשן נעבעך, טרויעריק און דערשלאָגן \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.972501</td>\n","      <td>קאָנגרעס ס נייַ פּאַלאַסיז געמאכט פילע מענטשן ...</td>\n","      <td>0</td>\n","      <td>positive</td>\n","      <td>[קאָנגרעס ס נייַ פּאַלאַסיז געמאכט פילע מענטשן...</td>\n","      <td>[-0.007056341972202063, -0.0033368950244039297...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.972501  ...  [-0.007056341972202063, -0.0033368950244039297...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":37}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"5h-pha_nPoBc","executionInfo":{"status":"ok","timestamp":1620202587337,"user_tz":-300,"elapsed":625626,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"a8a486c3-0557-4be3-b6a0-57d6daf6a218"},"source":["# Yiddish for: 'Congress's newest polices made many people happy '\n","fitted_pipe.predict(\"קאָנגרעס ס נייַ פּאַלאַסיז געמאכט פילע מענטשן צופרידן \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.999997</td>\n","      <td>קאָנגרעס ס נייַ פּאַלאַסיז געמאכט פילע מענטשן ...</td>\n","      <td>0</td>\n","      <td>positive</td>\n","      <td>[קאָנגרעס ס נייַ פּאַלאַסיז געמאכט פילע מענטשן...</td>\n","      <td>[0.002619848819449544, -0.018449869006872177, ...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.999997  ...  [0.002619848819449544, -0.018449869006872177, ...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":38}]},{"cell_type":"markdown","metadata":{"id":"XSz4WzScaAHj"},"source":["# The Model understands Kygrgyz\n","![ky](https://www.worldometers.info/img/flags/small/tn_kg-flag.gif)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"DXz6fhJSaAHu","executionInfo":{"status":"ok","timestamp":1620202587920,"user_tz":-300,"elapsed":626067,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"7b409dc7-7ec0-47c4-b42a-24e24b67c830"},"source":["# Kygrgyz for: 'Congress's newest polices made many people poor, sad and depressed '\n","fitted_pipe.predict(\"Конгресстин жаңы саясаты көптөгөн адамдарды жакыр, кайгыга чөгүп, көңүл чөгөттү \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.987505</td>\n","      <td>Конгресстин жаңы саясаты көптөгөн адамдарды жа...</td>\n","      <td>0</td>\n","      <td>negative</td>\n","      <td>[Конгресстин жаңы саясаты көптөгөн адамдарды ж...</td>\n","      <td>[-0.0002846009156201035, -0.002948872279375791...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.987505  ...  [-0.0002846009156201035, -0.002948872279375791...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":39}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"lh_ZSHlPaAHv","executionInfo":{"status":"ok","timestamp":1620202588449,"user_tz":-300,"elapsed":626583,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"5fc6a691-eb12-4cd2-e24c-6391ce070614"},"source":["# Kygrgyz for: 'Congress's newest polices made many people happy '\n","fitted_pipe.predict(\"Конгресстин жаңы саясаты көпчүлүктү кубандырды \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.999999</td>\n","      <td>Конгресстин жаңы саясаты көпчүлүктү кубандырды</td>\n","      <td>0</td>\n","      <td>positive</td>\n","      <td>[Конгресстин жаңы саясаты көпчүлүктү кубандырды]</td>\n","      <td>[0.018544858321547508, -0.003260228084400296, ...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.999999  ...  [0.018544858321547508, -0.003260228084400296, ...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":40}]},{"cell_type":"markdown","metadata":{"id":"DGMVMKaTdJFj"},"source":["# The Model understands Tamil\n","![ta](https://www.worldometers.info/img/flags/small/tn_in-flag.gif)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"JWDr_LoCdJFn","executionInfo":{"status":"ok","timestamp":1620202589857,"user_tz":-300,"elapsed":627966,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"9eb05327-e956-4fb5-a5ee-62dc31e5b372"},"source":["# Tamil for: 'Congress's newest polices made many people poor, sad and depressed '\n","fitted_pipe.predict(\"காங்கிரசின் புதிய கொள்கைகள் பலரை ஏழைகளாகவும், சோகமாகவும், மனச்சோர்வடையச் செய்தன \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.974183</td>\n","      <td>காங்கிரசின் புதிய கொள்கைகள் பலரை ஏழைகளாகவும், ...</td>\n","      <td>0</td>\n","      <td>negative</td>\n","      <td>[காங்கிரசின் புதிய கொள்கைகள் பலரை ஏழைகளாகவும்,...</td>\n","      <td>[-0.006572885904461145, -0.0003982966300100088...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.974183  ...  [-0.006572885904461145, -0.0003982966300100088...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":41}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"Q6C0BmTtdJFp","executionInfo":{"status":"ok","timestamp":1620202590214,"user_tz":-300,"elapsed":628312,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"02b09df5-298a-48f4-af0a-ba7bbf241971"},"source":["# Tamil for: 'Congress's newest polices made many people happy '\n","fitted_pipe.predict(\"காங்கிரசின் புதிய கொள்கைகள் பலரை மகிழ்ச்சியடையச் செய்தன \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","      <th>origin_index</th>\n","      <th>trained_sentiment</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_labse</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.999997</td>\n","      <td>காங்கிரசின் புதிய கொள்கைகள் பலரை மகிழ்ச்சியடைய...</td>\n","      <td>0</td>\n","      <td>positive</td>\n","      <td>[காங்கிரசின் புதிய கொள்கைகள் பலரை மகிழ்ச்சியடை...</td>\n","      <td>[0.018834874033927917, -0.01959705911576748, -...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   trained_sentiment_confidence  ...                           sentence_embedding_labse\n","0                      0.999997  ...  [0.018834874033927917, -0.01959705911576748, -...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":42}]},{"cell_type":"markdown","metadata":{"id":"2BB-NwZUoHSe"},"source":["# 5. Lets save the model"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"eLex095goHwm","executionInfo":{"status":"ok","timestamp":1620203583379,"user_tz":-300,"elapsed":1621468,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"bcc02088-1c38-4c16-d115-80708a73636c"},"source":["stored_model_path = './models/classifier_dl_trained' \n","fitted_pipe.save(stored_model_path)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Stored model in ./models/classifier_dl_trained\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"e_b2DPd4rCiU"},"source":["# 6. Lets load the model from HDD.\n","This makes Offlien NLU usage possible!   \n","You need to call nlu.load(path=path_to_the_pipe) to load a model/pipeline from disk."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"SO4uz45MoRgp","executionInfo":{"status":"ok","timestamp":1620204133344,"user_tz":-300,"elapsed":124905,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"2f75322c-c669-4956-e4ed-0388960ada0b"},"source":["stored_model_path = './models/classifier_dl_trained' \n","hdd_pipe = nlu.load(path=stored_model_path)\n","preds = hdd_pipe.predict('I am extremly depressed and down cause of school and just feel like ending my life...')\n","preds"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>origin_index</th>\n","      <th>sentiment_confidence</th>\n","      <th>sentence</th>\n","      <th>sentence_embedding_from_disk</th>\n","      <th>document</th>\n","      <th>sentiment</th>\n","      <th>text</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>8589934592</td>\n","      <td>[0.9989802, 0.9989802]</td>\n","      <td>[I am extremly depressed and down cause of sch...</td>\n","      <td>[[-0.029913833364844322, -0.03314201161265373,...</td>\n","      <td>I am extremly depressed and down cause of scho...</td>\n","      <td>[negative, negative]</td>\n","      <td>I am extremly depressed and down cause of scho...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   origin_index  ...                                               text\n","0    8589934592  ...  I am extremly depressed and down cause of scho...\n","\n","[1 rows x 7 columns]"]},"metadata":{"tags":[]},"execution_count":1}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"btTWUdsDNhfx","executionInfo":{"status":"ok","timestamp":1620204215401,"user_tz":-300,"elapsed":823,"user":{"displayName":"Gammer Otaku","photoUrl":"","userId":"18042713576744284398"}},"outputId":"108b339c-4dea-4586-de63-8e424e41e9f6"},"source":["hdd_pipe.print_info()"],"execution_count":null,"outputs":[{"output_type":"stream","text":["The following parameters are configurable for this NLU pipeline (You can copy paste the examples) :\n",">>> pipe['document_assembler'] has settable params:\n","pipe['document_assembler'].setCleanupMode('shrink')              | Info: possible values: disabled, inplace, inplace_full, shrink, shrink_full, each, each_full, delete_full | Currently set to : shrink\n",">>> pipe['sentence_detector@SentenceDetectorDLModel_c83c27f46b97'] has settable params:\n","pipe['sentence_detector@SentenceDetectorDLModel_c83c27f46b97'].setExplodeSentences(False)  | Info: whether to explode each sentence into a different row, for better parallelization. Defaults to false. | Currently set to : False\n","pipe['sentence_detector@SentenceDetectorDLModel_c83c27f46b97'].setStorageRef('SentenceDetectorDLModel_c83c27f46b97')  | Info: storage unique identifier | Currently set to : SentenceDetectorDLModel_c83c27f46b97\n","pipe['sentence_detector@SentenceDetectorDLModel_c83c27f46b97'].setEncoder(com.johnsnowlabs.nlp.annotators.sentence_detector_dl.SentenceDetectorDLEncoder@2a835f76)  | Info: Data encoder | Currently set to : com.johnsnowlabs.nlp.annotators.sentence_detector_dl.SentenceDetectorDLEncoder@2a835f76\n","pipe['sentence_detector@SentenceDetectorDLModel_c83c27f46b97'].setImpossiblePenultimates(['Bros', 'No', 'al', 'vs', 'etc', 'Fig', 'Dr', 'Prof', 'PhD', 'MD', 'Co', 'Corp', 'Inc', 'bros', 'VS', 'Vs', 'ETC', 'fig', 'dr', 'prof', 'PHD', 'phd', 'md', 'co', 'corp', 'inc', 'Jan', 'Feb', 'Mar', 'Apr', 'Jul', 'Aug', 'Sep', 'Sept', 'Oct', 'Nov', 'Dec', 'St', 'st', 'AM', 'PM', 'am', 'pm', 'e.g', 'f.e', 'i.e'])  | Info: Impossible penultimates | Currently set to : ['Bros', 'No', 'al', 'vs', 'etc', 'Fig', 'Dr', 'Prof', 'PhD', 'MD', 'Co', 'Corp', 'Inc', 'bros', 'VS', 'Vs', 'ETC', 'fig', 'dr', 'prof', 'PHD', 'phd', 'md', 'co', 'corp', 'inc', 'Jan', 'Feb', 'Mar', 'Apr', 'Jul', 'Aug', 'Sep', 'Sept', 'Oct', 'Nov', 'Dec', 'St', 'st', 'AM', 'PM', 'am', 'pm', 'e.g', 'f.e', 'i.e']\n","pipe['sentence_detector@SentenceDetectorDLModel_c83c27f46b97'].setModelArchitecture('cnn')  | Info: Model architecture (CNN) | Currently set to : cnn\n",">>> pipe['bert_sentence@labse'] has settable params:\n","pipe['bert_sentence@labse'].setBatchSize(8)                      | Info: Size of every batch | Currently set to : 8\n","pipe['bert_sentence@labse'].setCaseSensitive(False)              | Info: whether to ignore case in tokens for embeddings matching | Currently set to : False\n","pipe['bert_sentence@labse'].setDimension(768)                    | Info: Number of embedding dimensions | Currently set to : 768\n","pipe['bert_sentence@labse'].setMaxSentenceLength(128)            | Info: Max sentence length to process | Currently set to : 128\n","pipe['bert_sentence@labse'].setIsLong(False)                     | Info: Use Long type instead of Int type for inputs buffer - Some Bert models require Long instead of Int. | Currently set to : False\n","pipe['bert_sentence@labse'].setStorageRef('labse')               | Info: unique reference name for identification | Currently set to : labse\n",">>> pipe['sentiment_dl@labse'] has settable params:\n","pipe['sentiment_dl@labse'].setThreshold(0.6)                     | Info: The minimum threshold for the final result otheriwse it will be neutral | Currently set to : 0.6\n","pipe['sentiment_dl@labse'].setThresholdLabel('neutral')          | Info: In case the score is less than threshold, what should be the label. Default is neutral. | Currently set to : neutral\n","pipe['sentiment_dl@labse'].setClasses(['positive', 'negative'])  | Info: get the tags used to trained this SentimentDLModel | Currently set to : ['positive', 'negative']\n","pipe['sentiment_dl@labse'].setStorageRef('labse')                | Info: unique reference name for identification | Currently set to : labse\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"l_SDhKMysljL"},"source":[""],"execution_count":null,"outputs":[]}]}